Hypersparse Neural Network Analysis of Large-Scale Internet Traffic

The Internet is transforming our society, necessitating a quantitative understanding of Internet traffic. Our team collects and curates the largest publicly available Internet traffic data containing 50 billion packets. Utilizing a novel hypersparse neural network analysis of "video" streams of this traffic using 10,000 processors in the MIT SuperCloud reveals a new phenomena: the importance of otherwise unseen leaf nodes and isolated links in Internet traffic. Our neural network approach further shows that a two-parameter modified Zipf-Mandelbrot distribution accurately describes a wide variety of source/destination statistics on moving sample windows ranging from 100,000 to 100,000,000 packets over collections that span years and continents. The inferred model parameters distinguish different network streams and the model leaf parameter strongly correlates with the fraction of the traffic in different underlying network topologies. The hypersparse neural network pipeline is highly adaptable and different network statistics and training models can be incorporated with simple changes to the image filter functions.Comment: 11 pages, 10 figures, 3 tables, 60 citations; to appear in IEEE High Performance Extreme Computing (HPEC) 201

An analysis of the economic impact of strategic deaggregation

The work of Marcelo Bagnulo has been partially supported by project MASSES (TEC2012-35443) funded by the Spanish Ministry of Economy and Competitiveness (MINECO).The advertisement of more-specific prefixes provides network operators with a fine-grained method to control the interdomain ingress traffic. Prefix deaggregation is recognized as a steady long-lived phenomenon at the interdomain level, despite its well-known negative effects for the community. In this paper, we look past the original motivation for deploying deaggregation in the first place, and instead we focus on its aftermath. We identify and analyze here one particular side-effect of deaggregation regarding the economic impact of this type of strategy: decreasing the transit traffic bill. We propose a general Internet model to analyze the effect of advertising more-specific prefixes on the incoming transit traffic burstiness. We show that deaggregation combined with selective advertisements has a traffic stabilization side-effect, which translates into a decrease of the transit traffic bill. Next, we develop a methodology for Internet Service Providers (ISPs) to monitor general occurrences of prefix deaggregation within their customer base. Thus, the ISPs can detect selective advertisements of deaggregated prefixes, and thus identify customers which impact the business of their providers. We apply the proposed methodology on a complete set of data including routing, traffic, topological and billing information provided by a major Japanese ISP and we discuss the obtained results.Publicad

Unsupervised host behavior classification from connection patterns

International audienceA novel host behavior classification approach is proposed as a preliminary step toward traffic classification and anomaly detection in network communication. Though many attempts described in the literature were devoted to flow or application classifications, these approaches are not always adaptable to operational constraints of traffic monitoring (expected to work even without packet payload, without bidirectionality, on highspeed networks or from flow reports only...). Instead, the classification proposed here relies on the leading idea that traffic is relevantly analyzed in terms of host typical behaviors: typical connection patterns of both legitimate applications (data sharing, downloading,...) and anomalous (eventually aggressive) behaviors are obtained by profiling traffic at the host level using unsupervised statistical classification. Classification at the host level is not reducible to flow or application classification, and neither is the contrary: they are different operations which might have complementary roles in network management. The proposed host classification is based on a nine-dimensional feature space evaluating host Internet connectivity, dispersion and exchanged traffic content. A Minimum Spanning Tree (MST) clustering technique is developed that does not require any supervised learning step to produce a set of statistically established typical host behaviors. Not relying on a priori defined classes of known behaviors enables the procedure to discover new host behaviors, that potentially were never observed before. This procedure is applied to traffic collected over the entire year 2008 on a transpacific (Japan/USA) link. A cross-validation of this unsupervised classification against a classical port-based inspection and a state-of-the-art method provides assessment of the meaningfulness and the relevance of the obtained classes for host behaviors

A framework for alternate queueing: Towards traffic management by PC-UNIX based routers

Queueing is an essential element of traffic management, but the only queueing discipline used in traditional UNIX systems is simple FIFO queueing. This paper describes ALTQ, a queueing framework that allows the use of a set of queueing disciplines. We investigate the issues involved in designing a generic queueing framework as well as the issues involved in implementing various queueing disciplines. ALTQ is implemented as simple extension to the FreeBSD kernel including minor fixes to the device drivers. Several queueing disciplines including CBQ, RED, and WFQ are implemented onto the framework to demonstrate the design of ALTQ. The traffic management performance of a PC is presented to show the feasibility of traffic management by PC-UNIX based routers.

Managing Traffic with ALTQ

ALTQ is a package for traffic management. ALTQ includes a queueing framework and several advanced queueing disciplines such as CBQ, RED, WFQ and RIO. ALTQ also supports RSVP and diffserv. ALTQ can be configured in a variety of ways for both research and operation. However, it requires understanding of the technologies to set up things correctly. In this paper, I summarize the design trade-offs, the available technologies and their limitations, and how they can be applied to typical network settings. 1 Queueing Basics Essentially, every traffic management scheme involves queue management. A large number of queueing disciplines have been proposed to date in order to meet contradictory requirements such as fairness, protection, performance bounds, ease of implementation or administration. 1.1 Queueing Components Figure 1 illustrates queueing related functional blocks on a router. Each functional block could be needed to build a certain service but is not always required for other servi..